A new nonparametric approach for multiplicity control: Optimal Subset procedures

A new approach for multiplicity control (Optimal Subset) is presented. It is based on the selection of the best subset of partial (univariate) hypotheses which, when combined, produce the minimal p-value. The second step consists in multiplicity adjustment. In this work we show how to perform this new procedure in the permutation framework, choosing adequate combining functions and permutation strategies. The optimal subset approach can be very useful in exploratory studies because it performs a weak control for multiplicity which can be a valid alternative to the classic False Discovery Rate (FDR). A comparative simulation study and application to neuroimaging real data shows it is particularly useful in exploratory studies and when there is a high number of hypotheses. We also show how stepwise regression may be a special case of Optimal Subset procedures and how to adjust the p-value of the selected model taking into account the multiplicity arising from the various possible models selected by the stepwise regression.